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IHITLTEST READINESS REVIEW (TRR) 


• Jim Murphy 

• Mike Brignola 

• Conrad Rorie 

• Confesor Santiago 

• Mike Guminsky 

• Ken Cross 


Meeting Information 





• Location 

- AFRC, ISF 

• Date and Time 

- Friday, May 30, 2014 

- 8:00-12:00 PDT 

• Dial In Information 

- Number: 888-719-9352 

- Passcode: 4153130 

• WebEx 

- Meeting Number: 995 071 733 

- Meeting Password: UASNAS2014! 

- https://nasa.webex.eom/nasa/j. php?MTID=m291859e9efd510ee3be6eed0addlf4 
c7 
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IHITL Test Readiness Review Agenda 


• 8:00 Introduction (Murphy/Kim) -- 15 minutes 

o Objectives and Scope of this Review 
o Timeline (this is a high level schedule) 

• 8:15 Entrance/Exit Criteria (Brignola) -- 5 minutes 

• 8:20 FDR Summary (Brignola) - 10 minutes 

o RFIs 
o OCRs closed 

• 8:30 Overview & Objectives (Murphy) - 90 minutes 

o Summary of test goals and high level general objectives -- 15 minutes 
o HSI Review (Shively) - 45 minutes 

o SSI Ames Peer Experiment Review Summary (Santiago) - 15 minutes 
o SSI Langley Peer Experiment Review Summary (Guminsky) - 15 minutes 

• 10:00 Break - 15 minutes 

• 10:15 LVC-DE Status (Murphy/Kim) - 30 minutes 

o TestSetupl 

o Summary of V&V and Characterization Test Results 
o Test Setup 2 

o Summary of V&V and Characterization Test Results 
o Test Setup 3 (Langley brief) 
o Summary of V&V and Characterization Test Results 

o Includes a Langley chart (Murphy to provide a template) generated by Maria 
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IHITL Test Readiness Review Agenda 


• 10:45 Simulation Planning and Control Room Operations (Murphy) - 30 minutes 

o Test Plan (Data Analysis Plan) 
o Test Summary 

o Daily schedule 

o Seating plans 

o Test Coordination 

o Readiness status 

o Langley staffing 

o Armstrong staffing 

• 11:15 Safety Assessment (Ken Cross) - 5 minutes 

o Hazard Review 

• 11:20 Public Affairs- 5 minutes 

o Ames (Tran) 

o Armstrong (Maliska) 
o Langley (Schultz) 

• 11:25 Summary of Exit Criteria (Brignola) - 5 minutes 

• 11:30 Roll Call 
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Purpose of This Review /lEgl 



• Out of NPR 7123: A Test Readiness Review (TRR) 
ensures that the test article (hardware/software), test 
facility, support personnel, and test procedures are 
ready for testing and data acquisition, reduction, and 
control 

• The purpose of the this review is: 

a) To assess the readiness of the UAS NAS Technical team to 
perform the upcoming IHITL Simulation 

b) To assess whether the anticipated risks associated with 
conducting the tests are acceptable to the Project 
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Scope 


• The scope of this review covers content and conduct of the UAS-NAS 
Integrated Human In The Loop simulation 

• The review will focus on 

- Configuration of the Distributed Test Environment infrastructure developed to 
support the three IHITL Test Set-ups, including any known issues, limitations, or 
concerns 

- Readiness of the UASNAS Technical Team to support the IHITL Simulation by 
verifying that test LVC-DE is ready to support, support personnel, and test 
procedures are ready for testing and data acquisition, reduction, and control 

• This review will serve as the Experiment review for Test set-up 2 (Pilot 
Acceptability of SAA Maneuvers) 

- Summary review of Test Set-up 1 (Controller Acceptability of SAA Maneuvers) and 
Test set-up 3 (Acceptability of SAA Advisories) experiment reviews will be 
presented for background and context information 

• This review will serve as the SimLabs (Ames) Readiness Review 
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TRR Entrance / Exit Criteria 
FDR Closeout 


Mike Brignola 



7123.1b 



Entrance Criteria 

Success/Exit Criteria 

1. The objectives of the testing have been clearly 
defined and documented, and all of the test 
plans, procedures, environment, and 
configuration of the test item(s) support those 
objectives. 

2. Configuration of the system under test has been 
defined and agreed to. All interfaces have been 
placed under configuration management or have 
been defined in accordance with an agreed to 
plan, and a version description document has 
been made available to TRR participants prior to 
the review. 

3. All applicable functional, unit-level, subsystem, 
system, and qualification testing has been 
conducted successfully 

4. All TRR-specific materials, such as test plans, test 
cases, and procedures, have been available to all 
participants prior to conducting the review. 

5. All known system discrepancies have been 
identified and disposed in accordance with an 
agreed-upon plan. 

6. All previous design review success criteria and 
key issues have been satisfied in accordance with 
an agreed-upon plan. 

7. All required test resources/people (including a 
designated test director), facilities, test articles, 
test instrumentation, and other test enabling 
products/have been identified and are available to 
support required tests. 

8. Roles and responsibilities of all test participants 
are defined and agreed to. 

9. Test contingency planning has been 
accomplished, and all personnel have been 
trained. 

1. Adequate test plans are completed and approved 
for the system under test. 

2. Adequate identification and coordination of 
required test resources are completed 

3. Previous component, subsystem, and system test 
results form a satisfactory basis for proceeding 
into planned tests. 

4. Risk level is identified and accepted by 
program/competency leadership as required. 

5. Plans to capture any lessons learned from the test 
program are documented. 

6. The objectives of the testing have been clearly 
defined and documented, and the review of all 
the test plans, as well as the procedures, 
environment, and configuration of the test item, 
provides a reasonable expectation that the 
objectives will be met. 

7. Test cases have been reviewed and analyzed for 
expected results, and the results are consistent 
with the test plans and objectives. 

8. Test personnel have received appropriate training 
in test operation and safety procedures. 
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TRR Entry Status 


IHITL TRR 

Satisfied 

(Yes/No) 

Notes 

Entrance Criteria 

1 . The objectives of the testing have been clearly defined and 
documented, and all of the test plans, procedures, environment, and 
configuration of the test item(s) support those objectives. 

Y 

Objectives slide 14 
IHITL V&V Plan (5/2014) 

IHITL Data Analysis Plan (5/2014) 
IHITL Simulation Test Plan (5/2014) 
HSI Ames Experiment Review slide 17 
SSI Ames Experiment Review slide 54 
SSI LaRC Experiment Review slide 67 

2. Configuration of the system under test has been defined and 
agreed to. All interfaces have been placed under configuration 
management or have been defined in accordance with an agreed to 
plan, and a version description document has been made availa 

Y 

Final Design Review slide 10 
Document status slide 12 
IHITL system baseline CCB 4/2014 

3. All applicable functional, unit-level, subsystem, system, and 
qualification testing has been conducted successfully 

Y 

LVC verification 
RGCS verification 

Distributed Test Environment validation 

4. All TRR-specific materials, such as test plans, test cases, and 
procedures, have been available to all participants prior to conducting 
the review. 

Y 

posted on Knowledge Now: 

- WBS 5.1 IT&E\Reviews Mtgs & Presentations\ 
TRR 

Document status slide 12 

5. All known system discrepancies have been identified and disposed 
in accordance with an agreed-upon plan. 


Applicable Open DR's slide 11 
Applicable Open OCR's slide 11 

6. All previous design review success criteria and key issues have 
been satisfied in accordance with an agreed-upon plan. 


FDR closeout slide 10 (open) 

HSI Ames Experiment Review 
- approval in this brief slide 17 
SSI Ames Experiment Review slide 54 
SSI LaRC Experiment Review slide 67 

7. All required test resources/people (including a designated test 
director), facilities, test articles, test instrumentation, and other test 
enabling products/have been identified and are available to support 
required tests. 

Y 

IT&E ORD-01 (3/2014) 

IHITL Simulation Test Plan (5/2014) 
IHITL Configl weekly TIM (4/2-6/4/2014) 

8. Roles and responsibilities of all test participants are defined and 
agreed to. 

Y 

Test roles slide 36, 92-102 

9. Test contingency planning has been accomplished, and all 
personnel have been trained. 

Y 

Contingency staffing and testing 
Pilot confederate training (complete) 
Pilot subject training (upon arrival) 
Controller subject training (upon arrival) 
Test staff training slide 102, 114 
Control Room Safety Training slide 114 
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Final Design Review RFI Status & Closeout 


• Reference IHITL Delta DR RFI status as of 20140520 Excel sheet 

• 17 RFI's and 4 Observations from the IRT 

• To date: 

- 15 RFI's and 4 Observations answered and closed by originators 

- 2 RFI closures pending CONOPs update/baseline 

• Status Summary 

- Full RFI closure pending document updates: 

• CONOPs (POC: Sam Kim) 

- Risk = minimal with CONOPs communicated in FDR slides 

- Full FDR closeout pending: 

• RFI closure 

• Baseline of CONOPs and DTE ICD-01 (POC: Sam Kim) 

- Risk = minimal with draft Distributed Test Environment ICD available 

• Debra's approval 
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Items require closure prior to TRR. 


Open OCR’s 

5 / 27/14 




i i 
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Document Status 


Documents Required through IHITL 5/29/2014 



Document 

Doc# 

POC 

Baselined 

at 

SRR/SWRR 

/FDR 

STATUS/COMMENTS 

UAS/NAS Project Requirements Document 

UAS-PRO-1. 1-005-001 (May- 14) 

Dan Roth 

Jaseline Review 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=662514 


IT&E Concept of Operations 

IT&E Conops-01 

Sam Kim 

FDR 

DRAFT STARTED 


IT&E Objectives and Requirements Document 

IT&E ORD-01 (MARCH 14) 

Mike Brignola 

FDR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=652199 


IT&E ACAS-Xu ORD 

IT&E ORD-02 

Mike Brignola 

Jaseline Review 

NOT FOR IHITL-NEED ACAS 


IT&E Project Plan 

IT&E PP-01 (APR 14) 

Heather Maliska 

FDR 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=651633 


System Requirements Documents 

LVC SRD-01 REV B (MARCH 2014) 

Jamie Willhite 

SRR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=652223 


RGCS SRD-01 REV A (MARCH 2014) 

Kurt Sanner 

SRR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=653173 


UAS SRD-01 (JAN-14) 

Mike Marston 

SRR 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=620546 


Software Requirements Documents 

LVC SWRD-02 REV B (MARCH 2014) 

Jamie Willhite 

SWRR 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=652226 


RGCS SWRD-01 

Kurt Sanner 

FDR 

NOT FOR IHITL-NEED FT3 


Interface Control Documents 

LVC ICD-01 (IKHANA DATA READER) (AUG-13) 

Jamie Willhite 

FDR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=616729 


LVC ICD-02 (ADS-B INTERFACE) 

Jamie Willhite 

FDR 

NOT FOR IHITL-NEED FT3 


LVC ICD-03 (GATEWAY) REV A (MARCH-14) 

Srba Jovic 

FDR 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=660217 


DTE ICD- 01 (DISTRIBUTED TEST ENVIR) 

Mike Dandachy 

FDR 

DRAFT 

* 

RGCS ICD-01 (Intra RGCS) (APRIL 14) 

Kurt Sanner 

FDR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=653585 


RGCS ICD-02 (Inter RGCS) (APRIL 14) 

Kurt Sanner 

FDR 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=653587 


UAS ICD-01 

Mike Marston 

FDR 

NOT FOR IHITL-NEED FT3 


V&V Plans 

IHITL V&V-01 (May-14) 

Jim Murphy 

FDR 

https ://nsckn. nasa.gov/DMS/FolderDocs. as px?Filter=124&FolderlD=152717 

* 

LVC V&V-01 (JA 

Jamie Willhite 

FDR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=657757 


RGCS V&V-01 (May- 14) 

Kurt Sanner 

FDR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=664287 


UAS V&V-01 

Mike Marston 

FDR 

NOT FOR IHITL-NEED FT3 


RGCS IHITL Test Plan 

RGCS-IHITL-TP-01 (May-14) 

Kurt Sanner 

TRR 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=664286 


IHITL Simulation Test Plan 

IHITL SIM TP-01 Rev A (May-14) 

Neil Otto 

TRR 

https ://nsckn. nasa.gov/DMS/FolderDocs. as px?Filter=124&FolderlD=152717 

* 

Risk Management Plan 

IT&E RMP-01 (May- 14) 

Jen Hinckley 

FDR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=665161 


Configuration Management Plan 

IT&E CMP-01 Rev B (FEB-14) 

J. Weigelt 

SRR 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=629634 


Software Development Plans 

IT&E SDMP-01 (MAR 14) 

J. Murphy 

SWRR 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=643 147 


LVC SDP-01 (JAN-14) 

Jamie Willhite 

SWRR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=630232 


ADS-B SDP-01 (JAN-14) 

Ricardo Arteaga 

SWRR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=621802 


IHITL Data Analysis Plan 

IT&E DAP-01 (May-14) 

Kim/Murphy 

TRR 

https ://nsckn. nasa.gov/DMS/FolderDocs. as px?Filter=124&FolderlD=152717 

* 

Architecture Description Document 

IT&E ADD-01 (APRIL 14) 

Brignola/Saltzman 

FDR 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=660364 


Safety and Mission Assurance Plan 

IT&E SMA-01 (FEB 13) (REV B IN REVIEW) 

Cross 

FDR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=616893 


Mishap Preparedness and Contingency Plan 

IT&E MPC-01 (M, 

Cross 

FDR 

https://nsckn.nasa.gov/DMS/ViewDoc. as px?DoclD=637507 


LVC/RGCS Safety Plan 

IT&E LSP-01 (May- 14) 

Cross 

TRR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=665248 


LVC Software Design Description 

LVC SWDD-03 REV A (F 

Jovic 

FDR 

https://nsckn.nasa.gov/DMS/ViewDoc. aspx?DoclD=649053 


RGCS System Baseline 

RGCS SBL-01 REV A (MAY 14) 

Victor Loera 

FDR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=663417 


IT&E Product/Specification Tree 

IT&E PST-01 (DEC-13) 

Mike Brignola 

FDR 

https ://nsckn.nasa.gov/DMS/ViewDoc.aspx?DoclD=616891 


* = l JN5IGNFD COPY IS l OCATFD ON NSCKN- https://nsckn. nasa.gov/DMS/FolderDocs. aspx?Filter=124&FolderlD=152717 
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IHITL Objectives 
Schedule 


Jim Murphy 
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IHITL Goals and Objectives 


• Goals 

- Integrate and evaluate the state of UAS concepts and supporting technologies defined 
within the scope of the UAS in the NAS Project. Identify areas of future research and 
development emphasis and reduce risk for the flight tests and capstone event 

- Evaluate and measure the effectiveness and acceptability of the SAA algorithms and 
displays to inform and advise pilots of UAS aircraft and air traffic controllers 

• High Level Objectives 

- Integrate and evaluate the state of UAS concepts and supporting technologies defined 
within the scope of the UAS in the NAS Project as a function of traffic scenarios 

- Evaluate the pilot and controller acceptability of UA maneuvers in response to SAA 
advisories 

- Assess the impact of wind uncertainty on the execution of SAA advisories as well as 
interoperability with TCAS equipped aircraft 

- Assess the impact of communication delay and wind uncertainty on the air traffic 
controller workload and acceptability of SAA self-separation maneuvers. 

- Evaluate the acceptability and performance of the LVC to provide a relevant 
environment 

- Collect data to improve batch simulation models (Test Setup 1 and 2) 
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IHITL Timeline 




- Test Set-up 3 has 2 test conditions that run concurrently 

- Test Set-up 2 is taking a break during SARP in mid-June 

• Eating into Make-up days 

- Test Set-up 1 can extend into 30 June week if necessary 

• Not desired, Holiday week and prep for Test Set-up 2 


• Test Set-up 2 

— Make-up days 28 July week 
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IHITL Experiment Reviews 


• Experiment Reviews 

- Controller Acceptability of SAA Maneuvers (Test Setup 1) 

• Completed May 19 th 

• Summary to follow 

- Acceptability of SAA Advisories (Test Setup 3) 

• Completed May 15 th 

• Summary to follow 

- Pilot Acceptability of SAA Maneuvers (Test Setup 2) 

• Experiment Review to be conducted as part of this TRR 

• Conrad Rorie to present 
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National Aeronautics and Space Administration 



HSI iHitL Experiment Review 

Pilot Subjects ("Configuration 1, Test Setup 2") 

• 30 MAY, 2014 


Jay Shively, HSI PE 
Lisa Fern, HSI 
Conrad Rorie, HSI 
Confesor Santiago, SSI PE 
Eric Mueller, SSI 
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Overview 



• Objective(s) 

• Experimental Design 

• Procedure 

• Data Analysis 

• Roles and Responsibilities 

• Success Criteria 
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Objectives 



• Leverage the lessons learned from PT4 to continue the 
evaluation of candidate Sense and Avoid (SAA) Self-Separation 
(SS) and Collision Avoidance (CA) displays and algorithms 

- Examine the effects of advanced traffic display elements and tools on 
pilots' ability to maintain well clear 

• Tools selected based on preliminary and subjective reports from 
participants in PT4 

• Examine the effects of different sensor ranges on pilots' ability 
to maintain well clear 

— Unlike PT4, will model 2 separate sensors with markedly 
different performance characteristics 
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Objectives 



• Rationale 

- PT4 compared a 'Basic' display configuration to a suite of 'Advanced' 
features 

• iHitL design will allow us to measure any differences between specific 
display features and determine the implications of their presence or 
absence in the GCS 

- Non-cooperative traffic are a major consideration in the development 
of SAA systems 

• The presence of non-cooperative aircraft provides a more realistic 
environment 

• Limited surveillance range and field of regard may have significant impact 
on pilot performance and behavior 


20 



Objectives 



• Lessons Learned: 

- Based on observations during PT4 data collection, subjective 

questionnaires, and pilot debriefs, several improvements have been 
instantiated by AFRL & SSI: 

1. VSCS (AFRL) 

- Control and navigation interface made more intuitive 

- Aircraft responds more consistently with pilot expectations 

- Ineffective/unused Advanced features removed to reduce clutter 

2. AutoResolver/SAA Proc (SSI) 

- Algorithm logic modified to recommend fewer extreme maneuvers 

- 'Push point' calculation optimized to make trial planning tool more responsive and 
accurate 
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Objectives 



• Additional Functionality 

- SSI and IT&E were able to exploit an existing field in the LVC Gateway to 
identify whether a target is cooperative or non-cooperative 

- Those identified as non-cooperative will be picked up by simulated airborne 
radar (modeled on Partner airborne radar) 

• ADS-B Equipped 

- Range = 80nm & +/- 5000ft 

• Non-Equipped Aircraft 

- Range = 6nm 

- Azimuth = +/- HOdeg (from nose) 

- Elevation = +/- 20deg (from horizontal) 

- Non-cooperative will use same symbology as cooperative aircraft within 
Vigilant Spirit 

• May not be assigned a callsign 
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Experimental Design 



• One-Way Repeated Measures Factorial 

- Display Information Level (4 Level; Within Subjects) 

• Configuration 1: Advanced Display with Information Only 

• Configuration 2: Advanced Display with Information + Vector Planner 

• Configuration 3: Advanced Display with Information + Auto Resolutions 

• Configuration 4: Advanced Display with Information + Vector Planner + 

Auto Resolutions 

- Roughly same as 'Advanced' suite in PT4 

• Embedded Variable 

- Track Type (manipulated within each scenario; not counterbalanced) 

1. Cooperative Traffic (ADS-B-equipped) 

2. Non-Cooperative Traffic 

• Participants 

- 5 active California Air National Guard (163 rd Reconnaissance Squadron) MQ-9 (Reaper) pilots 

- 5 active Beale Air Force Base RQ-4 (Global Hawk) pilots 
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Experimental Design 









Display Configuration 
will be 

counterbalanced 
between participants 

Track Type (cooperative 
vs. non-cooperative) 
will be manipulated 
within each scenario 

Encounters 

• Plan to have 8 
encounters with 
ownship per run 

• 4 encounters will 

involve a non- 
cooperative AC 


Pilot# 

Trial 1 

Trial 2 

Trial 3 

Trial 4 


Display Configuration 

Display Configuration 

Display Configuration 

Display Configuration 

1 

1 

2 

4 

3 


Display Configuration 

Display Configuration 

Display Configuration 

Display Configuration 

2 

2 

3 

1 

4 


Display Configuration 

Display Configuration 

Display Configuration 

Display Configuration 

3 

3 

4 

2 

1 


Display Configuration 

Display Configuration 

Display Configuration 

Display Configuration 

4 

4 

1 

3 

2 


Display Configuration 

Display Configuration 

Display Configuration 

Display Configuration 

5 

1 

2 

3 

4 


Display Configuration 

Display Configuration 

Display Configuration 

Display Configuration 

6 

2 

3 

4 

1 


Display Configuration 

Display Configuration 

Display Configuration 

Display Configuration 

7 

3 

4 

1 

2 


Display Configuration 

Display Configuration 

Display Configuration 

Display Configuration 

8 

4 

1 

2 

3 


Display Configuration 

Display Configuration 

Display Configuration 

Display Configuration 

9 

1 

2 

3 

4 


Display Configuration 

Display Configuration 

Display Configuration 

Display Configuration 

10 

4 

3 

2 

1 
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Procedure 



Alert Information 

1. Data Tag launches 
automatically 

• Callsign (for 
ADS-B 

equipped AC) 

• Groundspeed 

• Altitude 

• Vertical 
Velocity 

2. Intruder color- 
coded based on 
predicted threat 
level 

3. Auditory alert 

1. "Traffic, 
Traffic" for SS 

2. "Left/Right/CIi 
mb/Descend" 
for CA 

4. "Traffic" tag 
presented below 
Baseball Card 

5. Color-coded halo 
appears around 
ownship 

6. Closest Point of 
Approach 
Location 


Display Configuration 1: 
Advanced Information 

HnK/ 
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Procedure 





Alert Information 

1. Data Tag launches 
automatically 

• Callsign (for ADS- 
B equipped AC) 

• Groundspeed 

• Altitude 

• Vertical Velocity 

2. Intruder color- 
coded based on 
predicted threat 
level 

3. Auditory alert 

1. "Traffic, Traffic" 
for SS 

2. "Left/Right/Clim 
b/Descend" for 
CA 

4. "Traffic" tag 
presented below 
Baseball Card 

5. Color-coded halo 
appears around 
ownship 

6. Closest Point of 
Approach Location 

7. Vector Probe 
available for use 


Display Configuration 2: 
Information + Vector 

Drrtho 
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Procedure 





Alert Information 

1. Data Tag launches 
automatically 

• Callsign (for ADS- 
B equipped AC) 

• Groundspeed 

• Altitude 

• Vertical Velocity 

2. Intruder color- 
coded based on 
predicted threat 
level 

3. Auditory alert 

1. "Traffic, Traffic" 
for SS 

2. "Left/Right/Climb 
/Descend" for CA 

4. "Traffic" tag 
presented below 
Baseball Card 

5. Color-coded halo 
appears around 
ownship 

6. Closest Point of 
Approach Location 


Display Configuration 3: 
Information + Auto 

Rocnli itinnc 



7. Auto Resolution 
provided to right of 
baseball card 
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Procedure 





Alert Information 

1. Data Tag launches 
automatically 

• Callsign (for ADS- 
B equipped AC) 

• Groundspeed 

• Altitude 

• Vertical Velocity 

2. Intruder color- 
coded based on 
predicted threat 
level 

3. Auditory alert 

1. "Traffic, Traffic" 
for SS 

2. "Left/Right/Climb 
/Descend" for CA 

4. "Traffic" tag 
presented below 
Baseball Card 

5. Color-coded halo 
appears around 
ownship 

6. Closest Point of 
Approach Location 

7. Vector Probe 
available for use 


Display Configuration 4: 
Information + Vector 

Drnho 4- A i itn Pocrtli it inn 



8. Auto Resolution 
provided to right of 
baseball card 
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Procedure 



Pilots will see all four display conditions throughout the day 
• Pilot Tasks: 

1. Fly HAWK21 along pre-filed path (Fire Line) 

1. Only responsible for navigating the aircraft 

2. Requires interaction with the GCS and coordination ATC 

2. Comply with ATC clearances and traffic display alerts to maintain safety of flight 

• 2 levels of alert (similar in structure to TCAS II): 

- Self Separation Alert (similar to TA) - pilot uses discretion in deciding if/how to maneuver 

- Collision Avoidance Alert (similar to RA) - pilot instructed to immediately comply with 
presented RA 

• Return to mission route and altitude as often as possible 

3. Monitor and respond to chat and health/status tasks 



* Research GCS Layout at NASA Armstrong* 
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Procedure 



• General Parameters 

- Flying Instrument Flight Rules 

• Standard separation requirements 

- 500ft vertical separation with VFR 

- 1000ft vertical separation with IFR 

• ATC has been provided with documentation of flight plan 

- Pre-approved for mission altitude only 

- Operating UAS for a portion of its flight 

• UAS/GCS will launch en route (at mission altitude) 

- No takeoff or landing 

• Appropriate flight plan will be pre-loaded 

- UAS will launch in transit from WP01 to WP02 


Deviations from mission route or altitude require 
coordination with ATC - unless there is a safety of flight concern 



Procedure 





UAS Scenario (38min) 

- UAS: Predator B 

- Airspace: ZOA 40/41 

• Class A & E with mixed IFR/VFR traffic 

• Manned traffic modeled using real traffic data 

• 2 separate traffic scenario files 

- Secondary tasks: 

• 'Chat' directed 

- "Mission" Radio Frequency changes 

- Radial and Distance check 

- Nominal information requests (e.g., fuel level remaining) 



• System Alerts 

- Generator Failure (checklist) 

- Tanker Header Overpressure (checklist) 

- Annunciator checks 
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Procedure 


Run Schedule 


Time 

Task 

Duration 

0800 

Introduction / Pilot Briefing 

20 

0820 

VSCS Training 

30 

0850 

VSCS Practice 

30 

0920 

Break 

10 

0930 

Advanced Display Training 

20 

0950 

Advanced Display Practice 

20 

1010 

System Reset 

5 

1015 

Advanced Display Data Collection 

40 

1055 

Post Trial Forms 

10 

1105 

Break 

10 

1115 

Advanced Display Training 

10 

1125 

Advanced Display Practice 

20 

1145 

System Reset 

5 

1150 

Advanced Display Data Collection 

40 

1230 

Post Trial Forms 

10 

1240 

Lunch 

60 

1340 

Advanced Display Training 

10 

1350 

Advanced Display Practice 

20 

1410 

System Reset 

5 

1415 

Advanced Display Data Collection 

40 

1455 

Post Trial Forms 

10 

1505 

Break 

10 

1515 

Advanced Display Training 

10 

1525 

Advanced Display Practice 

20 

1545 

System Reset 

5 

1550 

Advanced Display Data Collection 

40 

1630 

Post Trial Forms 

10 

1640 

Debrief 

20 

1700 

End 
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Procedure 



• Multi Aircraft Control System (MACS) 

- Stationed at NASA Ames 

- A retired Air Traffic Controller will be managing UAS and manned traffic within 
ZOA40/41 

• Sector traffic modeled using real sector activity and data 

- Pseudo-pilots monitoring and managing all manned traffic (IFR & VFR) 

- ATC and pseudo-pilots communicating over frequency 118.42 



Air Traffic Control Station (MACS) 
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Data Analysis 



Real Time Intruder Data: 

- Vertical Distance 

- Lateral Distance 

- Slant Range 

- Predicted Closest Point of Approach (CPA) 

- Slant range to CPA 

- Time to loss of well clear 

- Threat events 

Post Process Intruder Data: 

- Well Clear Violations 

- NMAC Violations 

- Time in threat levels 

- Min distance to intruder (vertical, lateral, 
slant) 

- Min CPA 

- Min distance to Well Clear Violation 
Subjective Questionnaires 

- NASA-TLX 

- Post Trial 

- Post Sim 


Pilot Performance, Primary Task (Post 
Process) 

- RT to threats (initial Initiate action) 

- Edit/input RT 

- Upload RT 

- Correctness of maneuver (compared to 
recommended maneuver) 

- Size of maneuver 

- Secondary threats 

- Comm Data 

Pilot Task Performance, Secondary Task (Post 
Process) 

- deviations from flight plan 

• RMS error from initial route 

• Time off route 

- RT/accuracy for chat tasks 

- RTto health and status alerts, etc. 

- RT/accuracy to measurement task 
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Data Analysis 



• Data will be analyzed using a One-Way Repeated Measures Factorial 
ANOVA 

- Pilot Measured Response 

• Time to Reply to ATC 

• Time to Initiate Edit 

• Time to Edit 

• Time to Upload 

- # of Well Clear Violations 

- Closest Point of Approach (CPA) 

- Size of Resolution Maneuver 

- Tool Use (did pilots use Vector Probe/Auto Resolver?) 
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Roles and Responsibilities 




Integration Check 

Pre-Test/Shakedown 

Final Test 

Location 

Staffing Requirements 

7-Jul-2014 

8-Jul-2014 

9-Jul-2014 

10-Jul-2014 

ll-Jul-2014 

Armstrong 

Researcher (Conrad) 

X 

X 

X 

X 

X 

Sim Manager (AFRC; Jamie) 

X 

X 

X 

X 

X 

Engineering Support (Victor?) 

X 

X 

X 

X 

X 

UAS Pilot 



NASA Pilot 

NASA Pilot 


Ames 

Sim Manager (ARC; Neil?) 

X 

X 

X 

X 

X 

Engineering Support (Mohamad?) 

X 

X 

X 

X 

X 

Controller (Confederate) 

X 

X 

X 

X 

X 

Ghost Controller (Wayne) 

X 

X 

X 

X 

X 

Ghost Pilot (Jacob) 

X 

X 

X 

X 

X 

Pseudo Pilot 1 (Confederate) 

X 

X 

X 

X 

X 

Pseudo Pilot 2 (Confederate) 

X 

X 

X 

X 

X 

Pseudo Pilot 3 (Confederate) 

X 

X 

X 

X 

X 

Pseudo Pilot Observer (Sabrina?) 

X 

X 

X 

X 

X 




Data Collection 

Location 

Staffing Requirements 

14-Jul-2014 

15-Jul-2014 

16-Jul-2014 

17-Jul-2014 

18-Jul-2014 

21-Jul-2014 

22-Jul-2014 

23-Jul-2014 

24-Jul-2014 

25-Jul-2014 

Armstrong 

Researcher (Conrad) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Sim Manager (AFRC; Jamie) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Engineering Support (On Call; Victor?) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

UAS Pilot 

ANG 

ANG 

ANG 

ANG 

ANG 

BEALE 

BEALE 

BEALE 

BEALE 

BEALE 

Ames 

Sim Manager (ARC; Neil?) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Engineering Support (On Call; Mohamad?) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Controller (Confederate) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Ghost Controller (Wayne) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Ghost Pilot (Jacob) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot 1 (Confederate) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot 2 (Confederate) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot 3 (Confederate) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 




• 7 JULY - 11 JULY will be testing and practice 

• 14 JULY - 25 JULY will be data collection 

• Researchers and confederates will be stationed at AFRC and ARC 

36 



Success Criteria 


• Generate at least half of the encounters in all four trials for 
each pilot participant 

- Allows HSI to analyze effects of Display Configuration and Track Type 
by studying rates of well clear violations 

• Pilot responds to well clear threats by interacting with GCS 
and coordinating/communicating with ATC 

- Allows HSI to record the pilots' measured response 

• Pilot completely fills out subjective questionnaires in a timely 
manner (within 20min of scenario completion) 

- Allows HSI to analyze impact of scenarios on participants level of 
acceptability and workload 




37 


Questions? 





• Division Approval? 
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BACKUP SLIDES 
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Test Setup 1 Experiment Summary 


Confesor Santiago 




Controller Acceptability of SAA Maneuvers (Test Setup 1) 



Objectives: 


• Evaluate the acceptability to the controller of 
maneuvers performed for "self-separation" in 
order to remain "well-clear" of other traffic 


• Collect data that improves modeling of controller 
performance in batch non-real-time simulations 


55 



Success Criteria 



• Evaluate the acceptability to the controller of maneuvers performed for 
"self-separation" in order to remain "well-clear" of other traffic 

- Procedure comparison in which SS maneuvers are or are not coordinated 
with ATC before execution is successful at informing whether coordination 
should be required (or not). 

- Maneuver and scripted deviation magnitude the controllers (1) detect and 
(2) object to are recorded and used to evaluate ATC acceptability. 

- Data to support DAA use conops for RTCA SC-228 Ops sub-group is collected. 


• Collect data that improves batch simulations 

- The time required for controllers to approve or disapprove self-separation 
maneuvers is collected and used to improve model. 

- Controller-approved maneuvers are recorded and used to improve self- 
separation algorithm recommendations. 

- Time and distance thresholds and airspace/scenario circumstances at which 
ATC issues traffic advisories are recorded and used to improve models. 


56 



Test Conditions 



• UAS pilot using ground control station (confederate) 

• Virtual manned and unmanned background traffic (confederate) 

- Mix of IFR and VFR (simulated cooperative and non-cooperative sensors) 

- Flown by pseudo-pilots 

• Single sector of interest (controller subject) 

- Fifteen recently retired air traffic controllers 

- Retired within last five years 

- Enroute or TRACON experience acceptable 

• Adjacent sectors staffed with confederate controller to: 

- Conduct handoffs 

- Ensure scripted encounters do not get perturbed and remain realistic 

• Scripted scenarios over ZOA (sector 40/41 combined) low-altitude airspace 

- Loss of "well-clear" encounters (10 per run) 

- Traffic density/workload (light-to-moderate, moderate, high) 

- Maneuver deviations for non-encounter aircraft 

• Three weeks of data collection, 5 days a week, one subject a day 

- June 9-27 

- Each controller subject will be trained morning of data collection 

- Participant scripts have been written and are ready 
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Experiment Review Outcome 



• Experiment Review - Monday, May 19, 2014 

— NASA Ames standard experiment review process for when human subjects are 
involved 

- Approximately 2 hours (briefing and Q&A) 

• Review Panel (Ames - Code AF) 

- Miwa Hayashi, Tom Kozon, Sandy Lozito, Savvy Verma, and Kathy Lee 

• Also in attendance... 

- Fellow AF colleagues 

- UAS-NAS subprojects and Project Office 

• Presented in-depth detail of experiment plan 

• Outcome 

- Approved to execute the experiment (mgmt and review panel) 

- No changes to experiment design 

- Numerous comments/suggestions 

- IRB completed 
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Experiment Review Feedback 


• Concerns about the number of simulated encounters 

— Two SME verified encounters are typical/realistic for given airspace 

- Encounters are counter-balanced across scenario 

- Encounter count falls within realistic ATC work-load 

• Concerns with scenario complexity assessment 

- Used SME to verify complexity assessment 

• Suggested additional situational awareness metric 

— Taken into account additional metric to record ATC subjective work-load estimates 
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Data Analyses 



• Observers will note whether the controller contacted the pilot or pointed out the 
deviation, and when traffic advisories were given 

- Observations will be scanned at the end of each day for data security 

- Data will be transcribed into electronic version, and processed using analysis tools (e.g. 
Matlab, Excel) 

• Questionnaires for the controller subjects will be administered at the end of each 
run 

- Questionnaires will be scanned at the end of each day 

- Subjective data will be transcribed into electronic version for later processing 

• Encounter-specific data (e.g. closest point of approach, conflict start time) will be 
recorded in log files 

- Logs files will be archived each day from the computer used to administer the 
experiment 

- Data will be analyzed with post-processing scripts using log files as direct input 

• At the end of this experiment, all data will be analyzed to achieve the 
aforementioned success criteria of the test objectives. 

- Lessons learned and modeling improvements will be drawn out and used to inform 
future simulations and flight tests 
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BACKUP SLIDES 
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Traffic Scenarios and Levels 


• Four scenarios derived from previous experiment (PT4) with modifications 

- Each scenario contains ten IFR-VFR aircraft "encounters" 

- Five encounters between UAS and VFR aircraft 

- Five encounters between manned IFR and VFR aircraft 

- UAS aircraft 

- Predator B as simulated by VSCS/RGCS 

- Range of other aero characteristics based on manned aircraft types 

- VFR traffic in three types 

- Receiving air traffic services with an operational transponder 

- Not receiving air traffic services but possessing an operational transponder 

- Not receiving air traffic services and not possessing a transponder ("primary target") 

- Traffic complexity determined by aircraft density and requirement to maintain 
a "miles-in-trail" restriction for aircraft entering NorCal TRACON 




Scenario 

Workload 

Complexity 

Additional Services Impacted 

C-l 

Moderate 

Not difficult 

Yes 

C-2 

Light to Moderate 

Not difficult 

Yes 

C-3 

Heavy 

Difficult 

Yes 

C-4 

Moderate 

Difficult 

Yes 
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Subjects 


• Fifteen recently retired air traffic controllers 

- Retired within last five years 

- Enroute or TRACON experience acceptable 

• Instructions 

- Issue TAs to appropriate aircraft as you see fit 

• Remind them that they should do so when "separation may reduce below the applicable 
standard, ... or place aircraft in unsafe proximity" 

• Primary-only targets should all be below 10,000 ft, but remember that issuing TAs to 
aircraft receiving air traffic services for all know intruders is an important additional 
service 

• [other standard ATC separation services also referenced during training] 

— To help us improve simulation fidelity, please point out any simulation or scenario 
inconsistencies, problems, concerns, bugs, etc. during the run. 

• The observer will note these and follow up with any additional details you wish to 
provide at the end of the run. 

• The pseudo-pilot adherence to real operations is important, please point out any time 
they don't conform to your expectations or it raises your workload/increases complexity. 
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Daily Schedule 



Time 

Task 

Duration 

830 

Introduction /Controller Briefing 

20 

850 

MACS Training 

30 

920 

MACS Practice 

80 

1040 

Break 

10 

1050 

First Scenario 

40 

1130 

Post Trial Forms 

10 

1140 

Lunch 

60 

1240 

Second Scenario 

40 

1320 

Post Trial Forms 

10 

1330 

Break 

10 

1340 

Third Scenario 

40 

1420 

Post Trial Forms 

10 

1430 

Break 

10 

1440 

Fourth Scenario 

40 

1520 

Post Trial Forms 

10 

1530 

Debrief 

60 


1630 End 
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Metrics 





• During run 

- Number of communications and time the pilot and controller spend discussing the 
encounter 

- Encounter-specific data 

• Losses of well clear 

• Closest point of approach 

• Maneuver type and deviation magnitude 

- Traffic advisory data 

• Time to CPA at which an advisory was issued 

• Predicted CPA that necessitated an advisory 

- Number of "objectionable" simulation artifacts 

• Whether a maneuver/procedure combination was noted 

• Whether a maneuver outside an encounter was noted 
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Metrics (cont.) 


• Post run 

- Workload and traffic complexity 

— Acceptability of aircraft flows, maneuvers 

- Selection of specific objectionable simulation "artifacts" 

• For example, which maneuver/procedure combinations were not preferred 

• Which maneuvers outside an encounter were noticed 
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Test Setup 3 Experiment Summary 


Mike Guminsky 


Acceptability of SAA Advisories (Test Setup 3) 


• Goal: Collection of performance metrics to determine SAA-TCAS interoperability and address 
the impact of winds and expected communication delay on the execution of self-separation 
tasks 

• Test Conditions (we'll describe the Sim architecture in later slides): 

- This experiment is broken into two separate data collection efforts 

- TCAS data collection 

• Manned aircraft (virtual) with TCAS installed 

- Canned flight path 

• U AS aircraft 

- Canned flight path (multiple) 

• Flight paths designed to provide specific encounters 

• Variable winds are used to support evaluation of wind uncertainty 

- 5,000 - 17,000 ft 

- Controller data collection 

• UAS pilot in GCS (confederate) 

- Advisories provided by SAA (Stratway+) 

• Virtual manned and unmanned background traffic (confederate) 

• Single sector of interest (controller subject) 

- Retired DFW controllers 

• Staffed adjacent sectors to conduct handoffs (confederate) 

• Scripted scenarios over ZFW airspace 

- Provide encounters 

- Provide workload 
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Acceptability of SAA Advisories (Test Setup 3) 


• Test Objectives 

- Evaluate whether the Stratway+ calculated well clear volumes and pilot guidance are 
sufficient to keep the UA clear of the intruder TCAS II Collision Avoidance Volumes 

• Success: Simulated encounters with the TCAS II equipped aircraft are successful in informing 
the SS well clear volume definition. 

- Evaluate whether the Stratway+ calculated well clear volumes and pilot guidance are 
properly accounting for wind direction and velocity to keep the UA well clear of the 
intruder and the TCAS II Collision Avoidance Volumes 

• Success: Stratway+ SS guidance effectively resolves conflicts with the designated HMD 

- Evaluate whether SAA Self Separation maneuvers are acceptable to air traffic controllers 
under realistic C2 delays 

• Success: Successfully gather ATC subjective metrics of acceptability with varying 
communication delays. 

- Evaluate whether realistic C2 delays and varying wind conditions impact SAA design 
requirements 

• Success: Successfully gather performance data to inform the design of the SS function with 
varying winds and communication delays. 

- Determine effect of realistic C2 delays on controller perceptions of safe and unsafe 
conditions 

• Success: Successfully gather ATC subjective metrics of acceptability with regard to flight safety. 
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Self Separation Bands Example 




7 % NM 

3 




Head-On Encounter: 

• Global Hawk (194 kts, 3 deg/sec turn) 

• Grey Eagle (108 kts, 1 deg/sec turn) 

• Declaration Time = 120 secs 

• "TCAS Bands" (20 sec tau, .35 nmi HMD) 

• Different Performance = Different Bands 
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Acceptability of SAA Advisories (Test Setup 3) 


• Experiment Review Outcomes 

- No changes to experiment design 

- Proceed with experiment 

- IRB completed 

• Test Conduct: 

- TCAS data collection 

• 9-27 June (no subjects) 

- Controller data collection 

• 6 subjects 

• 2 days per Controller subject 

• Each controller subject trained morning of first day 

• 9-10 June, 12-13 June, 23-24 June, 26-27 June, 1-2 July, 8-9 July 

• Participant scripts are have been carried over from UASCAS 1 

• Data Analysis 

- TCAS data analysis 

• 28 July to 30 September 

- Controller data collection 

• 28 July to 30 September 




72 



I-HITL Data Analysis -TCAS Data Collection Parameters 



VSTALT 

5_1 *BLKR 50 

VSTHDG 

5_1 *BLKR 50 

VSTALTR 

5_1 *BLKR 50 

VSTSPD 

5_1 *BLKR 50 

VSTLAT 

5_1 *BLKR 50 

VSTLON 

5_1 *BLKR 50 

VSTBRG 

5_1 *BLKR 50 

VSTRNG 

5_1 *BLKR 50 

VSTID 

* 

5_1 *BLI1 8,50 


TCAS TRAFFIC A/C ALTITUDE [FT] F7.2 
TCAS TRAFFIC A/C HDG [DEG] F7.2 
TCAS TRAFFIC A/C ALT RATE [FT/S] F7.2 
TCAS TRAFFIC A/C SPD [KNTS] F7.2 
TCAS TRAFFIC A/C LATITUDE [DEG] F7.2 
TCAS TRAFFIC A/C LONGITUDE [DEG] F7.2 
TCAS A/C BRG REL TO SIM. A/C [DEGS] F7.2 
TCAS A/C SLANT RNG WRT SIM. A/C [NM] F7.2 
TCAS TRAFFIC A/C ID [-] 


VSTTRAF 5_1 *BLKR 50 TCAS TRAFFIC ACTIVE FLAG TRUE 

* 


SUTAONLY 5_1 *INT4 2 
SUCCLC 5_1 *LOGl .F. 
SUCCLN 5_1 *LOGl .F. 
SUCCRC 5_1 *LOGl .F. 
SUCOFC 5_1 *LOGl .F. 
SUDCRD 5_1 *LOGl .F. 
SUDDED 5_1 *LOGl .F. 
SUDDEN 5_1 *LOGl .F. 
SUICLM 5_1 *LOGl .F. 
SUIDES 5_1 *LOGl .F. 
SUMVES 5_1 *LOGl .F. 
SUMVES2 5_l*LOGl.F. 
SURCLB 5_1 *LOGl .F. 
SURDES 5_1 *LOGl .F. 
SUTCSFL 5_1 *LOGl .F. 
SUTCSOK 5_1 *LOGl .F. 
SUTTR 5_1 *LOGl .F. 
SUTTR2 5 1 *LOGl .F. 


TCAS in TA only mode, no RAs perm. (P) 19 
Aurl warn "CLIMB, CLIMB, CLIMB" (CAE) TRUE 
Aurl warn "CLIMB, CLIMB NOW" (CAE) TRUE 
Aurl warn "CLIMB, CROSSING CLIMB"(CAE) TRUE 
Aurl warn "CLEAR OF CONFLICT" (CAE) TRUE 
Aurl warn "DESCEND, CROSSING DESCEND" (CAE) TRUE 
Aurl warn "DESCEND, DESCEND, DESCEND" (CAE) TRUE 
Aurl warn "DESCEND, DESCEND NOW"(CAE) TRUE 
Aurl warn "INCREASE CLIMB" (CAE) TRUE 
Aurl warn "INCREASE DESCENT" (CAE) TRUE 
Aurl warn "MONITOR VERTICAL (CAE) TRUE 
Aurl warn SUMVES twice (CAE) TRUE 
Aurl warn "REDUCE CLIMB" (CAE) TRUE 
Aurl warn "REDUCE DESCENT" (CAE) TRUE 
Aurl warn "TCAS SYSTEM TEST FAIL"(CAE) TRUE 
Aurl warn "TCAS SYSTEM TEST OK" (CAE) TRUE 
Aurl warn "TRAFFIC, TRAFFIC" (CAE) TRUE 
Aurl warn "TRAFFIC, TRAFFIC" (CAE) TRUE 
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I-HITL Data Analysis - Flight State Data Parameters 


MsgFlightState Message: 

recordedUtcTime, m_acid, m_cid, m_timeCreated,m_timeReceived, 
mJatitude,mJongitude,mjDressureAltitude,m_geoAltitude, 
m_indicatedAirSpeed,m_mach,m_bankAngle,m_pitchAngle, 
m_groundSpeed,m_verticalSpeed, 

m_trueHeading,m_magneticVariation,m_trueGroundTrack,m_trueAirSpeed, 

m_altitudeTarget,m_headingTarget,m_speedTarget,m_verticalSpeedTarget, 

m_equipageFlags,m_modeFlags,m_dlnkFlags,m_configurationFlags, 

m_flaps,m_speedBrake,m_windDirection,m_windSpeed,m_outerAirTemperature, 

m_mapRangeCaptain,m_mapRangeFo,m_headingBug, 

m_vhfFrequency,m_beaconCode,m_geoSectorld,m_atcSectorld, 

m_acSectorld,m_atcSectorName 
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LVC Status and V&V Testing 


Jim Murphy 



IHITL Test Set-up 1 



• Ames 

- Controllers 

• MACS 

- Pseudo Pilots 

• MACS 

- HLA 

• IEEE 1516 
standard Pitch 
portable (RTI) 

• Armstrong 

- GCS (UAS pilot) 

- RGCS developed 
for this project 

- VSCS: AFRL 
software 

• SAA: 
AutoResolver/ 
CA 

• Traffic Display 

• Voice Comm: 

- Plexsys: DIS- 
based 
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I HITL Test Set-up 2: 



• Ames 

- Controllers 

• MACS 

- Pseudo Pilots 

• MACS 

- HLA 

• IEEE 1516 
standard Pitch 
portable (RTI) 

• Armstrong 

- GCS (UAS pilot) 

- RGCS developed 
for this project 

- VSCS: AFRL 
software 

• SAA: 
AutoResolver/ 
CA 

• Traffic Display 

- Eye Tracker 

• Voice Comm: 

- Plexsys: DIS- 
based 
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IHITL Test Set-up 1 & 2 


• V&V Results 

— System Verified May 15-16 

- System Validated May 21-29 

• LVC infrastructure was tested to characterize the distributed message passing latencies 

- Avg Latency = 24 ms 

- Stdev = 14 ms 

- Deemed acceptable by Ames SSI and HSI teams 

• Data from dry run was collected and reviewed by Researchers for completeness 

- Deemed acceptable for IHITL to proceed 

• Traffic scenarios have been reviewed by researchers 

- Deemed acceptable for IHITL data collection 

• IHITL system approved by Ames SSI for use for IHITL (Test Setup 1) 

- Ames HSI waiting for resolution of data drop before approving IHITL system (see 
Issues slide) 

- Testing Procedures 

• Validation Test: LVC-TPR-08 RevB 

• Delta Software Test Procedure: LVC-TPR-07 
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IHITL Test Set-up 1 & 2: Validation Matrix 


IT&E Research and Technical Objectives 


Req ID 


Keyboard and 
Mouse Control for 
Pilot Station 


Voice 

Communication 
Latency with ATC 


RGCS Human 
Factors 


Short Name 


The RGCS pilot control station 
incorporates Human Input 
Devices (keyboard and mouse) 
for data entry and vehicle control. 


Characterize voice 
communication latencies 
between the RGCS pilot and 
virtual air traffic controllers 
(record and timestamp). 


Develop an operationally 
representative UAV GCS in 
support of the Distributed Test 
Environment for simulation and 
Flight Test. 


Integrate Vigilant 
Spirit CS 


Objective 

from IT&E ORD-OI 3/1/2014 


IT&E 

HSI 


Integrate relevant subprojects 
control stations into the RGCS 
(Vigilant Spirit). 


Stake- 

holder 


To LVC Req 


v Flow Down v 


SS 
Allocation 


RGCS 

LVC 



Keyboard, mouse and 
keypad are present, 
integrated with dfrgcs- 
vscs. Mouse is available 
for dfrgcs-csd. 


Manufacturer-provided 
system latency 


LVC Validation Test 
(LVC-TPR-08) 
supported by LVC 
Verifications 


Description of 
Validation 


All keys pressed are 
recognized by VSCS 
software, mouse 
movement for VSCS and 
CSD are acceptable to 
users, no unintended 
consequences due to 
key presses, mouse 
movement moves curser 
in direction of display 
location (move mouse 
off-screen right to move 
the cursor to the next 
display located on the 
right of the current 
display) 


Estimated latency falls 
within researcher 
criteria. 


LVC Validation Test 
(LVC-TPR-08) 
supported by LVC 
Delta-SW Verification 
Test (LVC-TPR-07) 


Success Criteria 


All keys pressed are recognized by VSCS software, 
mouse movement for VSCS and CSD are 
acceptable to users, no unintended consequences 
due to key presses, mouse movement moves 
curser in direction of display location (move 
mouse off-screen right to move the cursor to the 
next display located on the right of the current 
display) 


Researchers have accepted estimate latency. 


System deemed 
acceptable by all 
participants. 

LVC IHITL 
Verifications 
complete, l. The RGCS 
pilot is able to 
communicate with 
virtual ATC at Ames 

2. The RGCS pilot is able 
to control the 
(simulated) UAV in 
accordance to the 
directions given by ATC. 

3. The RGCS sends state 
information to the LVC 
environment. 

4. The RGCS collects and 
displays other aircraft in 
the relevant airspace 
generated within the 
LVC environment. 

5. The RGCS generates 
the appropriate 
indications resulting 
from the SSI algorithm 
alert messages. 


Demonstration of VS 
ntegration and 
operation. 

Delta-SW Verification 
complete. 


Summary Results 


1 . Pilot communication with ATC and all 
other connected participants is clear. PTT 
handset and foot pedals both work. 

2. Pilot commands given to the UAV in all 
autopilot modes are all accepted and the 
appropriate response from the UAV 
(simulation) is observed. 

3. The RGCS UAV (Hawk-21) state (position, 
heading, etc) is observed within the LVC 
environment. 

4. The RGCS air traffic displays on both CSD 
and within the VS TSD match what is 
generated and sent through the LVC network. 

5. The RGCS warnings and cautions audible 
and visible alerts are generated by SSI 
algorithm message inputs to the RGCS. 


Display of air traffic on Vigilant Spirit and 
CSD has been judged acceptable by HSI and 
SSI. 

Vigilant Spirit operates as expected (with 
state update rates and within the limitations 
of the dfrgcs-vscs computer specifications). 

Results from LVC Validation and LVC Delta- 
SW Verification testing demonstrate 


Pass / 
Fail 


Explanation of any 
Non-Conformance 
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IHITL Test Set-up 1 & 2: Validation Matrix 


IT&E Research and Technical Objectives 


Req ID 


Display LVC air 
traffic on HSI 
displays 


IT&E test 
efficiency and re- 
configurability 


Short Name 


Display air traffic data from the 
LVC infrastructure on the HSI 
provided air traffic displays 
(CSD). 


Provide a multi-use RGCS and 
LVC system allowing for flexible 
configurations through modular 
design (hardware / software), 
multi-function equipment and 
multi-function software. 


IT&E 

Characterization 
of the Distributed 
Test Environment 
for Simulation and 
Flight Test 


IT&E Record, 
Distribute and 
Manage Data 


Integrate UAS in 
the NAS 
subproject 
technologies 


Objective 

from IT&E ORD-OI 3/1/2014 


HSI 

SSI 


Provide the capability in the 
RGCS and LVC to support the 
IT&E characterization of the 
Distributed Test Environment for 
simulation and flight test. 


Record, distribute and manage 
research data of the Distributed 
Test Environment for simulation 
and flight test 


Demonstrate the integration of 
the UAS in the NAS subproject 
technologies (HSI, SSI, Comm) 
using simulations and flight test 
(IHITL, FT3&FT4). 


Stake- 

holder 


To LVC Req 


IT&E 

HSI 

SSI 

Comm 


IT&E 

HSI 

SSI 

Comm 


v Flow Down v 


SS 
Allocation 


RGCS 

LVC 


RGCS 

LVC 


LVC-10,1 1 , 
19 


LVC- 

1.2. 4. 5. 6. 9.1 

2.13.14.16.1 
9,20 


LVC- 

1,2,4,5,7,16 


RGCS 

LVC 


RGCS 

LVC 


RGCS 

LVC 

SUA 



LVC Validation Test 
(LVC-TPR-08) 
supported by LVC 
Delta-SW Verification 
Test (LVC-TPR-07) 


LVC Validation Test 
(LVC-TPR-08) 
supported by LVC 
Delta-SW Verification 
Test (LVC-TPR-07) 


Description of 
Validation 


Researchers judge 
system acceptable. 

Delta-SW Verification 
complete. 

CSD displays air traffic 
data generated by MACS 
(when used). The VSCS 
built-in air traffic display 
displays the air traffic 
generated by MACS 
when used. 


Demonstration of a 
fully integrated and 
functioning system. 


Data Analysis from 
LVC Validation Test 
(LVC-TPR-08) 


Data Management 
results from LVC 
Validation Test (LVC- 
TPR-08) 


LVC Validation Test 
(LVC-TPR-08) 
supported by LVC 
Verifications 


Success Criteria 


Display of air traffic on Vigilant Spirit and 
CSD has been judged acceptable by HSI and 
SSI. 

Results from LVC Validation and LVC Delta- 
SW Verification testing demonstrate 
performance. 


Results from LVC Validation and LVC Delta- 
SW Verification testing confirm system 
integration and performance. 


Validation Test results 
judged acceptable 
from both technical 
and research 
perspectives. 


Effective data 
management 
demonstrated. VSCS 
data files successfully 
recorded, sent to 
Ames for post-test 
analysis 


Succ e ssfu l I H I TL 
Vigilant Spirit suite 
successfully integrated 
nto the RGCS, SSI 
algorithms 
successfully integrated 
nto the LVC with 
resulting displays and 
audio integrated into 
RGCS 


Summary Results 


Average Latency = 24 ms 
Standard Deviation = 14 ms 

Results deemed acceptable by Ames SSI 
and HSI teams. 


Data exchange capabilities [(AFRC - 
and (LaRC -ARC)] in place. 


ARC) 


Data from LVC Validation tests has been 
judged acceptable by HSI and SSI. 


VS Suite successfully integrated into RGCS, 
SSI messages successfully integrated into 
RGCS (through VS software and CSD) 


Pass / 
Fail 


TBO 

Pass 


Explanation of any 
Non-Conformance 


Further data 
exchanges planned to 
exercise the data 
exchange process 
during Rehearsal 
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IHITL Test Set-up 1 & 2 


• Issues, Limitations, or Concerns 

- Eye Tracker Software has not yet installed 

• It is only needed for Test setup 2 and is expected to be installed and tested prior to the 
July data collection time 

- EyeVis has not yet been installed 

• It is only needed for Test setup 2 and is expected to be installed and tested prior to the 
July data collection time 

- Startup procedures take too long 

• We continue to practice start-up of the system 

• It is critical that this be done in a timely manner to ensure enough time to perform the 
planned data collection 

• Routine start-ups are 4-5 minutes, since each data collection effort is proceeded by a 
break, 10 minutes should be sufficient for IHITL 

— Network Latencies 

• During testing on 27-28 May data drop and large latencies were observed 

• These are under investigation 

- Automated data archive not fully tested 
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I HITL Test Set-up 1 & 


• Contingency Planning 

- Eye Tracker Software has not yet installed 

• Forfeit eye tracking data collection 

- EyeVis has not yet been installed 

• Install Camtasia as backup 

- Network Latencies 

• Possible software patch 

- Add message header flag to more easily re-sync after lost data packet 

- Does not solve problem, but mitigates its impact 

• Delay Test Setup 2 for two weeks 

- Provides opportunity to test and debug 

• Run GCS out of Ames FDDRL or DSRL 

- This is not desired since distributed testing is an I H ITL goal 

- However, may be necessary for Test Setup 2 where the pilot is the subject 

• Delay and move 

- Debug for two weeks, if problem still not resolved, move to FDDRL or DSRL 

- Last Resort 

• Decision Point 

- 6/6/2014 (Friday before Test setup 1 data collection) 

- Automated data archive not fully tested 

• Manual archiving and access tested, scripts to be tested prior to IHITL 
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IHITL Test Set-up 3 



• Ames 

- Flight Simulator with TCAS II 

• B747 

• TCAS II version 7.0 

- HLA 

• IEEE 1516 standard Pitch 
portable (RTI) 

• Langley 

- GCS (UAS pilot) 

- Pilot Monitor 

• MACS 

- ATC Monitor 

• MACS 

• GCS 

- MACS GCS Engineering 
Display 

• Voice Comm: 

- SimPhonics: DIS-based 


TCAS Data Collection 


ARC CVSRF 


sap 


NA 74 
Toolbox 


HLA - High Level Architecture 


ARC DSRL 

1 

t 

n 

LVC GW Toolbox 




j 


LaRC SGT UAS LAB 


LVC 

Gateway 





Air Traffic Monitor 











IHITL Test Set-up 3: Controller Data Collection 





• Langley 

- GCS (UAS pilot) 

- Controllers 

• MACS 

- Pseudo Pilots 

• MACS 

- Pilot Monitor 

• MACS 

- ATC Monitor 

• MACS 

• GCS 

- MACS GCS 
Engineering 

• Voice Comm: 

- SimPhonics: DIS-based 

• UAS Comm delay is 
modeled 


LaRC SGT UAS LAB 


Voice Comm 
(Delay Model) 


MACS GCS 
CDTI + Stratway+ 


MACS GCS 
Monitor 


T 


MACS 

Sim Manager 
Traffic 
Generator 


MACS Pseudo Pilot 


MACS Pseudo Pilot 


1 


Air Traffic Monitor 


Air Traffic Controller 


Air Traffic Ghost 
Air Traffic Ghost 


V *5 













IHITL Test Set-up 


• V&V Results 

- System Verified May 15-16 

- System Validated May 21-29 

• LVC infrastructure was tested to characterize the distributed message passing latencies 

- Avg Latency = 47 ms 

- Stdev=6ms 

- Deemed acceptable by Langley SSI team 

• Data from dry run was collected and reviewed by Researchers for completeness 

- Controller subject data collection validated 

- TCAS data validation happening today (based on Wednesday tests) 

• Traffic scenarios have been reviewed by researchers 

- Deemed acceptable for IHITL data collection 

• IHITL system approved by Langley SSI for use for IHITL 

- Testing conducted primarily out of NASA Langley 

• Testing with NASA Ames for TCAS conducted each Wednesday and Friday 

- Need to Board the IHITL versions of the software prior to data collection 

• Issues, Limitations, or Concerns 

- Network Latencies 

• The observed data drop between Ames and Armstrong may also impact TCAS data collection 

• This has not been observed, but we are monitoring 

- Automated data archive not fully tested 

• Contingency Planning 

- Network Latencies 

• Postpone TCAS data collection until problem is resolved 

- Automated data archive not fully tested 

• Manual archiving and access tested, scripts to be tested prior to IHITL 
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Test Planning 


Jim Murphy 


IHITL Simulation Planning 


• Test Plan 

- Complete and signed (5/27/2014) 

- Includes 

• Description of LVC and simulation components 

• Test Management and Staffing 

• Test Schedule 

• Description of Pre and Post Test Brief 

• Description of Test and Objectives 

- Measurements and data collection 

• Test Procedures 

• Data Analysis Plan 

- Complete and signed (5/27/2014) 

- Details of measurements and data collection 

- Describes data archive and distribution plan 

• Each day, all data from each center and test will be sent to Ames for consolidation and archive 

• Archive includes 

- Test procedures run for that day (including any red-lines) 

- Versions of software used 

- Traffic scenarios 
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IHITL Simulation Planning 


Data Archiving: Test Setup 1 


Scenarios 


Software 



m 

i 


Ihitl s-1 


Configuration 


I 


IQtl 

Ul 


Notes 




runl 


run2... 


runl 


vscs 


gw 

1 

SaaProc 


macs 


voice 


Messaging in/Ou 
.cmf ties 




Data logger csv 
At 


i— 

L 

Log Fit .CSV 


MacsOutput 
log pies 


camtasa 

.aviFits 



StarxjjT) Tasks | ^ Sutoyswi 
.av U Amunonor av 



M< Kty vlf I 
oa Fits 



Ghost Plot 
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IHITL Simulation Planning 


Data Archiving: Test Setup 2 


Ihitl s-2 




runl 


run2... 



runl 


run2... 


gw 


SaaProc 1 




Data Logge' 
«• 

csv 

L 

Log Me .csv 


macs 


voice 


MacsOutput og 
fta 


Subsystem 
Annuncator Cjy 


VC Messages : £ye T'acke*' F es 


M*c Kay vsf and 
w «• s 
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IHITL Simulation Planning 



• Data Archiving: Test Setup 3 



90 



























IHITL Simulation Planning 


• Test Report to be written for FY14 API 

- Includes conduct of test 

- Characterization of system 

- Lessons Learned to inform future simulations and flight tests 

• Lessons Learned to be capture at the daily out-brief and included in Final Test Report 
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IHITL Simulation Planning 


• Key Personnel 




Name 

Organization 

E-mail 

Phone Number 

Jim Murphy 

NASA Ames 

jim.murphy@nasa.gov 

734.676.1164 

Sam Kim 

NASA Armstrong 

sam.k.kim@nasa.gov 

661.276.2036 

Jay Shively 

NASA Ames 

robert.j.shively@nasa.gov 

650.604.6249 

Confesor Santiago 

NASA Ames 

confesor.santiago@nasa.gov 

605.604.1916 

Maria Consiglio 

NASA Langley 

maria.c.consiglio@nasa.gov 

757.864.2651 

Jim Griner 

NASA Glenn 

jgriner@nasa.gov 

216.433.5787 

Neil Otto 

SAIC 

neil.d.otto@nasa.gov 

650.604.4604 

Eric Mueller 

NASA Ames 

eric.mueller@nasa.gov 

605.604.3529 

Srba Jovic 

SAIC 

srba.jovic@nasa.gov 

650.604.2116 

Diane Carpenter 

SAIC 

diane.m.carpenter@nasa.gov 

650.604.0699 

Lisa Fern 

OSU 

lisa.fern@nasa.gov 

650.213.6930 

Conrad Rorie 

SJSU 

conrad.rorie@nasa.gov 

650.604.5451 

Dimitri Tsakpinis 

LITES 

dimitrios.tsakpinis@nasa.gov 

757.224.4059 

Mike Guminski 

NASA Langley 

michael.guminsky@nasa.gov 

757.664.9579 

Jamie Willhite 

NASA Armstrong 

jamie.w.willhite@nasa.gov 

661.276.2198 

Kurt Sanner 

NASA Armstrong 

kurt.sanner-1 @nasa.gov 

661.276.2535 

Betty Silva 

NASA Ames 

betty.silva@nasa.gov 

650.604.2117 


Daily Schedule 

- See next charts 
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IHITL Test Setup 1: Daily Schedule 



Time 

Task 

Duration 

830 

Introduction /Controller Briefing 

20 

850 

MACS Training 

30 

920 

MACS Practice 

80 

1040 

Break 

10 

1050 

First Scenario 

40 

1130 

Post Trial Forms 

10 

1140 

Lunch 

60 

1240 

Second Scenario 

40 

1320 

Post Trial Forms 

10 

1330 

Break 

10 

1340 

Third Scenario 

40 

1420 

Post Trial Forms 

10 

1430 

Break 

10 

1440 

Fourth Scenario 

40 

1520 

Post Trial Forms 

10 

1530 

Debrief 

60 


1630 End 
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IHITL Test Setup 1: Staffing 





S 

iakedown/Dress Rehearsal 

Location 

Staffing Requrements 

Personnel 

2-Jun 

3-Jun 

4-Jun 

5-Jun 

6-Jun 

Armstrong 

LVC Lead Engineer 

Jamie Willhite 

X 

X 

X 

X 

X 

LVC Support 

Martin Hoffman 

X 

X 

X 

X 

X 

RGCS Support 

Gayle Patterson 

X 

X 

X 

X 

X 

Voice Support 

Marlin Hoffman 

X 

X 

X 

X 

X 

UAS Pilot 

FRA 

X 

X 

X 

X 

X 

Data Archiving 


X 

X 

X 

X 

X 

Ames 

Researcher 

Confesor Santiago 

X 

X 

X 

X 

X 

Test Conductor 

Neil Otto 

X 

X 

X 

X 

Paul Fast 

DSRL Lead Engineer 

Srba Jovic 

X 

X 

X 

X 

X 

Voice Comm Support 

Riva Canton 

X 

X 

X 

X 

X 

Sim Support 

Mohomad 

X 

X 

X 

X 

X 

Controller 

FRA 

X 

X 

X 

X 

X 

Ghost Controller 

Wayne Bridges 

X 

X 

X 

X 

X 

Ghost Pilot 

Jacob Pfeiffer 

X 

X 

X 

X 

X 

Pseudo Pilot 1 (Confederate) 

FRA 

X 

X 

X 

X 

X 

Pseudo Pilot 2 (Confederate) 

FRA 

X 

X 

X 

X 

X 

Pseudo Pilot 3 (Confederate) 

FRA 

X 

X 

X 

X 

X 

Pseudo Pilot Observer 

Sabrina? 

X 

X 

X 

X 

X 

Data Archiving 

Dave Brown 

X 

X 

X 

X 

X 
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IHITL Test Setup 1: Staffing 



Data Collection 


Location 

Staffing Requrements 


9-Jun 

10-Jun 

11-Jun 

12-Jun 

13-Jun 

16-Jun 

17-Jun 

18-Jun 

19-Jun 

20-Jun 

23-Jun 

24-Jun 

25-Jun 

26-Jun 

27-Jun 

Armstrong 

LVC Lead Engineer 

Jamie Willhite 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LVC Support 

Martin Hoffman 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

RGCS Support 

Gayle Patterson 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Voice Support 

Marlin Hoffman 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

UAS Pilot 

FRA 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Data Archiving 


X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Ames 

Researcher 

Confesor Santiago 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Test Conductor 

Neil Otto 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

DSRL Lead Engineer 

Srba Jovic 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Voice Comm Support 

Riva Canton 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Sim Support 

Mohomad (on call) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Controller 

FRA 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Ghost Controller 

Wayne Bridges 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Ghost Pilot 

Jacob Pfeiffer 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot 1 (Confederate) 

FRA 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot 2 (Confederate) 

FRA 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot 3 (Confederate) 

FRA 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot Observer 

Sabrina? 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Data Archiving 

Dave Brown 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 
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IHITL Test Setup 2: Daily Schedule 


Time 

0800 

0820 

0850 

Proposed Schedule 

Task 

Introduction / Pilot Briefing 
VSCS Training 
VSCS Practice 

Duration 

20 

30 

30 

0920 

Break 

10 

0930 

Advanced Display Training 

20 

0950 

Advanced Display Practice 

20 

1010 

System Reset 

5 

1015 

Advanced Display Data Collection 

40 

1055 

Post Trial Forms 

10 

1105 

Break 

10 

1115 

Advanced Display Training 

10 

1125 

Advanced Display Practice 

20 

1145 

System Reset 

5 

1150 

Advanced Display Data Collection 

40 

1230 

Post Trial Forms 

10 

1240 

Lunch 

60 

1340 

Advanced Display Training 

10 

1350 

Advanced Display Practice 

20 

1410 

System Reset 

5 

1415 

Advanced Display Data Collection 

40 

1455 

Post Trial Forms 

10 

1505 

Break 

10 

1515 

Advanced Display Training 

10 

1525 

Advanced Display Practice 

20 

1545 

System Reset 

5 

1550 

Advanced Display Data Collection 

40 

1630 

Post Trial Forms 

10 

1640 

Debrief 

20 

1700 

End 
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I HITL Test Setup 2: Staffing / Roles & Responsibilities 





Integration 

Check 

Pre-Test/Shakedown 


Location 

Staffing Requrements 

Personnel 

7-Jul 

8-Jul 

9-Jul 

10-Jul 

11-Jul 


Researcher 

Conrad Rorie 

X 

X 

X 

X 



LVC Lead Engineer 

Jamie Willhite 

X 

X 

X 

X 



LVC Support 

Martin Hoffman ? 

X 

X 

X 

X 


Armstrong 

RGCS Support 

Gayle Patterson 

X 

X 

X 

X 



Voice Support 

Martin Hoffman 

X 

X 

X 

X 



UAS Pilot 

AFRC Staff Pilot 



Dave Fedors 

Karl Magnusson 



Data Archiving 


X 

X 

X 

X 



Researcher 

Confesor Santiago 

X 

X 

X 

X 



Test Conductor 

Paul Fast 

X 

X 

X 

Srba Jovic 



DSRL Lead Engineer 

Srba Jovic 

X 

X 

X 

X 

Off Day 


Voice Comm Support 

Riva Canton 

X 

X 

X 

X 


Sim Support 

Mohamad 

X 

X 

X 

X 



Controller (Confederate) 

FRA 

X 

X 

X 

X 


Ames 

Ghost Controller 

Wayne Bridges 

X 

X 

X 

X 



Ghost Pilot 

Jacob Pfeiffer 

X 

X 

X 

X 



Pseudo Pilot 1 (Confederate) 

FRA 

X 

X 

X 

X 



Pseudo Pilot 2 (Confederate) 

FRA 

X 

X 

X 

X 



Pseudo Pilot 3 (Confederate) 

FRA 

X 

X 

X 

X 



Pseudo Pilot Observer 

Sabrina? 

X 

X 

X 

X 



Data Archiving 

Dave Brown 

X 

X 

X 

X 
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I HITL Test Setup 2: Staffing / Roles & Responsibilities 




Data Collection 

Location 

Staffing Requrements 


14-Jul 

15-Jul 

16-Jul 

17-Jul 

18-Jul 

21-Jul 

22-Jul 

23-Jul 

24-Jul 

25-Jul 

Armstrong 

Researcher 

Conrad Rorie 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LVC Lead Engineer 

Jamie Willhite 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LVC Support 

Martin Hoffman ? 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

RGCS Support 

Gayle Patterson 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Voice Support 

Martin Hoffman 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

UAS Pilot 

SME (subject) 

CAANG 

CAANG 

CAANG 

CAANG 

CAANG 

4th RW 

4th RW 

4th RW 

4th RW 

4th RW 

Data Archiving 


X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Ames 

Researcher 

Confesor Santiago 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Test Conductor 

Neil Otto 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

DSRL Lead Engineer 

Srba Jovic 

X 

X 

X 

Jeff H 

Jeff H 

Jeff H 

Jeff H 

Jeff H 

Jeff H 

Jeff H 

Voice Comm Support 

Riva Canton 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Sim Support 

Mohamad (On Call) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Controller (Confederate) 

FRA 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Ghost Controller 

Wayne Bridges 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Ghost Pilot 

Jacob Pfeiffer 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot 1 (Confederate) 

FRA 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot 2 (Confederate) 

FRA 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot 3 (Confederate) 

FRA 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pseudo Pilot Observer 

Sabrina? 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Data Archiving 

Dave Brown 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 
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IHITL Test Setup 3: Daily Schedule: TCAS Data Collection 



Time (in EDT) 

Task 

Duration (minutes) 

1130 

T-0 Test Team Pre-Brief 

30 

1200 

Data collection 

180 

1500 

Debrief 

30 

1530 

End 
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I HITL Test Setup 3: Staffing: TCAS Data Collection 



Location 


Ames 


Staffing Requrements 

Personnel 

2-Jun 

3-Jun 

4-Jun 

5-Jun 

6-Jun 

Test Conductor 

Ghislain Saillant 

X 

X 

X 

X 

Off 

Voice Support 

Dan Wilkins/Riva Canton 

N/A 

N/A 

N/A 

N/A 

HLA Support 

Jeff Hernandez 

X 

X 

X 

X 

B747 Pilot 

Sim Labs Staff 

N/A 

N/A 

N/A 

N/A 

Data Archiving 

Dave Brown 

X 

X 

X 

X 

Researcher 

Maria Consiglio 

X 

X 

X 

X 

Test Director 

Mike Guminsky 

X 

X 

X 

X 

Test Conductor 

Dimitri Tsakpinis 

X 

X 

X 

X 

LVC Support 

Dimitri Tsakpinis 

X 

X 

X 

X 

Pilot Monitor 

Mike Guminsky 

X 

X 

X 

X 

Controller Monitor 

Ray Comstock 

X 

X 

X 

X 


Langley 


Data Collection 


Location 

Staffing Requrements 


9-Jun 

10-Jun 

11-Jun 

12-Jun 

13-Jun 

16-Jun 

17-Jun 

18-Jun 

19-Jun 

20-Jun 

23-Jun 

24-Jun 

25-Jun 

26-Jun 

27-Jun 

Ames 

Test Conductor 

Ghislain Saillant 

X 

X 

X 

X 

Off 

X 

X 

X 

X 

Off 

X 

X 

X 

X 

Off 

Voice Support 

Dan Wilkins/Riva Canton 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

X 

X 

X 

X 

HLA Support 

Jeff Hernandez 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

B747 Pilot 

Matt or Gordon 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

X 

X 

X 

X 

Data Archiving 

Dave Brown 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Langley 

Researcher 

Maria Consiglio 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Test Director 

Mike Guminsky 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Test Conductor 

Dimitri Tsakpinis 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

LVC Support 

Dimitri Tsakpinis 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Pilot Monitor 

Mike Guminsky 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Controller Monitor 

Ray Comstock 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 
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HITLTest Setup 3: Daily Schedule: Controller Data Collectio 


Time (in EDT) 

Task 

Duration (minutes) 

Day 1 

Day 1 

Day 1 

0800 

T-0 Test Team Pre-Brief 

30 

0830 

Introduction / Pilot Briefing 

30 

0900 

Training 

90 

1030 

Data Collection 

60 

1130 

Post Run Evaluations 

30 

1200 

Lunch 

60 

1300 

Setup 

30 

1330 

Data Collection 

60 

1430 

Post Run Evaluations 

20 

F330 

Break 

10 

rsoo 

Data Collection 

60 

rain 

Post Run Evaluations 

30 

1630 

Debrief 

30 

r700 

End 


Day 2 

Day 2 

Day 2 

osoo 

T-0 Test Team Pre-Brief 

50 

0930 

Data Collection 

60 

T030 

Post Run Evaluations 

30 

TO50 

Break 

10 

1100 

Data Collection 

60 

1200 

Post Run Evaluations 

30 

1230 

Lunch 

60 

1330 

Setup 

30 

1400 

Data Collection 

60 

1500 

Post Run Evaluations 

30 

1530 

Debrief 

30 

1600 

End 
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I HITL Test Setup 3: Staffing: Controller Data Collection 





Shakedown/Dress Rehearsal 

Location 

Staffing Requrements 

Personnel 

2-Jun 

3-Jun 

4-Jun 

5-Jun 

6-Jun 


Researcher 

Maria Consiglio 


X 

X 




Test Director 

Jim Chamberlain 


X 

X 




Test Conductor 

Dimitri Tsakpinis 


X 

X 




LVC Support 

Dimitri Tsakpinis 


X 

X 




Pilot Monitor 

Jim Chamberlain 


X 

X 



Langley 

Controller Monitor 

Ray Comstock 

Off 

X 

X 

Off 

Off 

Controller (Subject) 

SME 

X 

X 


Ghost Controller 

SME 


X 

X 




Ghost Controller 

SME 


X 

X 




Pseudo Pilot 1 (Confederate) 

SME 


X 

X 




Pseudo Pilot 1 (Confederate) 

SME 


X 

X 




GCS Pilot (subject) 

SME 


X 

X 




Data Collection 


Location 

Staffing Requrements 


9-Jun 

10-Jun 

11-Jun 

12-Jun 

13-Jun 

23-Jun 

24-Jun 

25-Jun 

26-Jun 

27-Jun 

30-Jun 

1-Jul 

2-Jul 

8-Jul 

9-Jul 


Researcher 

Maria Consiglio 

X 

X 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 


Test Director 

Jim Chamberlain 

X 

X 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 


Test Conductor 

Dimitri Tsakpinis 

X 

X 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 


LVC Support 

Dimitri Tsakpinis 

X 

X 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 


Pilot Monitor 

Jim Chamberlain 

X 

X 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 

Langley 

Controller Monitor 

Ray Comstock 

X 

X 

Off 

X 

X 

X 

X 

Off 

X 

X 

X 

X 

X 

X 

X 

Controller (Subject) 

SME 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 


Ghost Controller 

SME 

X 

X 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 


Ghost Controller 

SME 

X 

X 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 


Pseudo Pilot 1 (Confederate) 

SME 

X 

X 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 


Pseudo Pilot 1 (Confederate) 

SME 

X 

X 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 


GCS Pilot (subject) 

SME 

X 

X 


X 

X 

X 

X 


X 

X 

X 

X 

X 

X 

X 
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IHITL Simulation Planning 


• Readiness Status 

- All personnel are under contract 

- Facilities are booked and ready for data collection 

- V&V complete 

- Software loaded and ready 

- Non-subject participants trained 

• Subjects are trained morning of data collection 

• Participant script have been prepared and are ready 
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Hazard Analysis 


Ken Cross 
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Current Hazard Analysis Status 



• Total IT&E Hazard Reports: 19 

• Total IHTL-applicable Hazard Reports: 1 
— Detailed on next slide 

• All hazard mitigations are complete & verified 
— No "Accepted Risk" hazards 
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Safety Procedures 



• Control Room/Facility Safety Plans: 

- AFRC: 

• IT&E RGCS/LVC Lab Safety Plan 

• IT&E Visitor Safety Checklist / Export Control Notice 

• Aircrew Flight Ops Manual DCP-O-025 (crew rest, etc.) 


• Ground Control Station Design for UAS in the NAS 

— Oversight by the Institutional Review Board for use of 
human subjects in research ops (crew rest, etc.) 

• Building Emergency Action Plans N243 & N257 


- ARC: 


- LaRC: 


• (on request) 


106 


1 IHTL-Applicable Hazard Report 





HR#: IT&E-18, Control Room Hazards 


Causes 

Effects 

Mitigations 

1 . Earthquake 

2. Emergency evacuation 

3. Exposure to energized equipment 

4. Exposure to sharp objects 

5. Exposure to hot surfaces 

6. Tripping hazards 

7. Spilled liquid/food on Control Room hardware 
system 

8. Extended exposure to cold temperatures in the 
RGCS/LVC Lab 

9. Lack of training/written procedures 

10. Over exposure to Inergen fire extinguishing agent in 
enclosed area 

11. Fatigue 

12. Lack of safety equipment (e.g., fire extinguisher, 
fire alarm pull station, emergency lighting, etc.) 

13. Equipment or building fire 

a. Injury to personnel 

b. Damage to hardware 

1) Require periodic independent safety inspections by Dryden 
Code SH and monthly inspections by team personnel 

2) Verify applicable hardware is compliant with earthquake 
safety requirements 

3) Develop and maintain a Control Room Safety Plan that is 
mandatory reading for all RGCS/LVC Lab operations 
personnel and includes personnel training requirements 

4) Train employees that food/liquid must be kept well clear of 
electronic equipment and to use spill proof containers to the 
extent possible 

5) Recommend personnel dress accordingly for the colder-than- 
normal temperatures in the Control Room 

6) Require access control to RGCS and LVC Lab 

7) Enforce crew rest requirements (per DCP-O-025) for 
designated “safety critical” personnel 

8) Design and build electronic systems to appropriate standards 
for personnel safety 

9) Verify employees have training in and quick access to fire 
extinguishers, fire alarm pull stations, evacuation route 
instructions, and other appropriate safety material 

10) Verify employees are trained and aware that the Inergen fire 
extinguishing system poses no threat to personnel when the 
system is activated and an emergency evacuation is ordered. 


Hazard Description 

Given that the project is tasked with designing and building a flight research 
control room consisting of some energized electrical equipment, it is possible 
that hazards may exist to personnel (project personnel and visitors) and 
potential damage to high-dollar hardware may be a consideration. This HR 
applies to both the Research Ground Control Station (RGCS) and the LVC 
Lab in B4840. 


Current Hazard Category 



A 

B 

C 

D 

Probability 

E A 

B 

C 

D 

E 


Cat I 


r 



m 



Cat 11 

y/y/FA 



W/j 


-M 

Cat III 


2 




u 



z 


QJ 

> 

QJ 

Cat IV 

□ 

□ 



□ 







<Z3 



Human 



Asset / Mission 







Current Hazard Categories 



National Aeronautics and 
Space Administration 
Armstrong Flight Research Center 


Human Safety 
Hazard Action Matrix (HAM) 
Residual Risk 


Probability [Pr] Categories 


Injury Severity 
Classifications 


I: Catastrophic 


A: Expected to 
occur 

(Pr > 10' 1 ) 


D: Unlikely to 


E: Improbable 
to occur 

(10* > Pr) 




Requires Center Director approval and may require approval by a higher authority. These hazards are defined as "Accepted Risks." 


Risk acceptance requires Center Director approval. These hazards are defined as "Accepted Risks" 


Risk acceptance requires Project Manager approval. 


TEM-OOIa 


108 



Current Hazard Categories 



National Aeronautics and 
Space Administration 
Armstrong Flight Research Center 


Loss of Asset/Mission 
Hazard Action Matrix (HAM) 
Residual Risk 


Asset/Mission 

Severity 

Classifications 


Probability [Pr] Categories 


A: Expected to 
occur 

(Pr> 10 1 ) 


B: Probable to 
occur 

(10 1 >Pr> 10 2 ) 


C: Likely to 
occur 

(10 2 > Pr> 10 3 ) 


D: Unlikely to 
occur 

(10 3 > Pr> 10*) 


E: Improbable 
to occur 

(10* > Pr) 


I: Catastrophic 


II: Critical 


III: Minor 


IV: Negligible 




Requires Center Director approval and may require approval by a higher authority. These hazards are defined as "Accepted Risks". 




Risk acceptance requires Center Director approval. These hazards are defined as "Accepted Risks". 


Risk acceptance requires Project/Program Manager approval. 


TEM-001b 
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Public Affairs and Export Control 


Due Tran 
Heather Maliska 
Vince Schultz 
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Open CCRs/DRs 



DR 

CCR 

STR 

Title 

Assigned To 

Statu 

s 

Plan for Closure 

13-103 

13-232 

13-324 

Ames MACS Trafic Not Displayed on DFRC MACS 

Hernandez 

Open 

Paperwork cleanup - Will close out of 
board. 


14-237 

14-327 

Build/Install New SAA Process Host Comps 

Code ME 

Open 

This has been completed, paperwork 
needs cleaning up. - Will close out of 
board. 


14-253 

14-335 

Install Plexcomm S/W on RGCS 

Hoffman 

Open 

Tested on all but Mission Director 
Station (not required for IHITL). Final 
testing to be conducted 5/30, but not 
necessary for IHITL. 


14-255 

14-336 

Install Plexcomm S/W on LVC 

Hoffman 

Open 

Tested on all but Mission Director 
Station (not required for IHITL). Final 
testing to be conducted 5/30, but not 
necessary for IHITL. 


14-260 

14-340 

Update Ikhana LVC Messages 

Patterson 

Open 

Not needed for IHITL 


14-269 


Install the required PLEXComm software on LVC Computers in 
support of the Plexsys Virtual Radio installation 

Hoffman 

Open 

Paperwork cleanup - Will close out of 
board. 

14-109 



Vigilant Spirit screen loses display of intruders for brief periods 
during siumlation runs 

Jovic 

Open 


CCB needed next week to: 

• Baseline DTE-ICD (open item from FDR) - open and close a 
new CCR 

• Baseline LaRC Test Set-up 3 

• Close DR-109 

Paperwork clean up needed 

Open, but not needed for 
IHITL 

Open 
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Public Affairs 




• Planned Tours 

— Ames 16-20 June (during SARP) 

• Non-interference 

• Confesor to give briefing 

• Visits between data collection 

• Planned demos 

- Various facilities 28 July - 1 August 

• May need to de-conflict if we need make-up days 

• Hands-on 

- VIP Viewing area included in LVC lab 

- Dryden TV 



n 

2 



Export Control (AFRC) 



VSCS Software code used in RGCS treated as ITAR 

- Displays are unrestricted, except for the electronic checklist. 

- Physical and network access should be secure and controlled 

- Access to U.S. Citizens and green card holders only, no foreign nationals 

- Anyone directly interacting with the VSCS software code (with the exception of research 
participants) should be supporting the UAS-NAS project as per the software agreement 

• Complete the training and return the signed certificate to Gayle Patterson. 

- Kurt 

- Victor 

- Ben 

- Bob Novy 

- Do not transmit VSCS information (code, images, video, spec, etc.) on the internet other than 
through approved, secure channels. 

• The preferred channels, in order are: 

- VDL (controlled by AFRL) 

- AMRDEC safe site (controlled by the Army) 

- NASA's File Transfer 

- Any information, including pictures, about VSCS that will be transmitted to the public is required to 
go trough NASA's public release process and vetted by AFRL (via the HSI subproject). 

- Visual demonstrations of the VSCS displays are unrestricted. 

• Hands-on interaction with the VSCS controls are restricted to project approved personnel 
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Training (AFRC) 




• Voice Comm Training 

- Who? 

• Test team on headset 

- What? 

• Use of Plexsys system, comm brevity, and protocol 

- When? 

• 6/2/14 

• IHITL Training 

- Who? 

• Those entering the LVC/RGCS (all participants) & visitors 

- What? 

• Briefing on Export Control 

• Control Room Safety Checklist 

• Visitors to sign in upon entry to the RGCS lab 

- When? 

• Test Team: 5/30/14 

• Visitors: to be briefed by escort upon arrival 

• Ames/Langley Training 

- Ames test members following existing SimLabs and PT4 protocols 

- Langley test members following existing UASCAS 1 protocols 
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Summary of Exit Criteria 


Mike Brignola 


Summary of TRR Exit Criteria 




IHITLTRR 

Satisfied 

^ 

Notes 

Success/Exit Criteria 

(Yes/No) 

1. Adequate test plans are completed and approved 
for the system under test. 


IHITL V&V Plan (5/2014) 

IHITL Data Analysis Plan (5/2014) 
IHITL Simulation Test Plan (5/2014) 

2. Adequate identification and coordination of 
required test resources are completed 


Test roles slide 36, 91-101 

3. Previous component, subsystem, and system test 
results form a satisfactory basis for proceeding into 
planned tests. 


PE's approval of Distributed Test 
Environment validation (slide 102): 

Kim, Murphy, Shively, Santiago, Consiglio, 
Griner 

4. Risk level is identified and accepted by 
proqram/competency leadership as required. 


Debra Randall's approval to proceed into 
test 

5. Plans to capture any lessons learned from the 
test program are documented. 


Reporting and lessons learned captured by 
Kim/Murhpy and individual PE's in IHITL 
daily reports 

6. The objectives of the testing have been clearly 
defined and documented, and the review of all the 
test plans, as well as the procedures, environment, 
and configuration of the test item, provides a 
reasonable expectation that the objectives will be 
met. 


TRR and documentation complete 

- IHITL CONOPs 

- IHITL Distributed Test Environment ICD 

7. Test cases have been reviewed and analyzed for 
expected results, and the results are consistent with 
the test plans and objectives. 


HSI Ames Experiment Review slide 1 7 
SSI Ames Experiment Review slide 54 
SSI LaRC Experiment Review slide 67 

8. Test personnel have received appropriate training 
in test operation and safety procedures. 


Pilot confederate training (complete) 
Pilot subject training (upon arrival) 
Controller subject training (upon arrival) 
Test staff training slide 101 , 113 
LVC/RGCS Safety Training slide 1 13 
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TRR Advice: Don't take the obvious for granted 




M 


